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Abstract. Let G„ !r (K) be the Grassmannian manifold of fc-dimensional K-subspaces 
in K n where K = R, C, H is the field of real, complex or quaternionic numbers. We 
consider the Radon, cosine and sine transforms, !Z r ',r, C r \ r and S r ', r , from the L 2 
space L 2 (G n ,r(^)) to the space L 2 (G„ jr '(K)), for r,r' < n — 1. The L 2 spaces are 
decomposed into irreducible representations of G with multiplicity free. We compute 
the spectral symbols of the transforms under the decomposition. For that purpose 
we prove two Bernstein- Sato type formulas on general root systems of type BC for 
the sine and cosine type functions on the compact torus M. r /2irQ v generalizing our 
recent results for the hyperbolic sine and cosine functions on the non-compact space 
W . We find then also a characterization of the images of the transforms. Our re- 
sults generalize those of Alesker-Bernstein and Grinberg. We prove further that the 
Knapp-Stein intertwining operator for certain induced representations is given by the 
sine transform and we give the unitary structure of the Stein's complementary series 
in the compact picture. 



1. Introduction 



The present paper is a continuation of our earlier papers [26] and [22] on Radon 
and related transforms and their relation to the Bernstein-Sato type formulas. We 
shall give here a rather unified approach to the Radon, cosine and sine transforms on 
Grassmannian manifolds. Let K = R, C, HI be the field of real, complex or quater- 
nionic numbers, and G n , r = Gr nr (K) be the Grassmannian manifold of fc-dimensional 
subspaces over K in K n . For l<r<r'<n — 1, v > 0, the Radon, cosine and sine 
transforms K = ft r> ,C = C^J,S = Sffl : C°°(G n , r ) -> C°°(G n y) are defined, for 
r] G G n y, by 



;i-2) (Cf)(v)= I |Cos(£,77)| 2 7(£K£, 

** Gn,r 

;i-3) (Sf)(v)= [ |Sin(£,77)| 2 <7(£K£, 



where d v £ is certain probability measure on the set {£ G G n ^ r : £ C rj} invariant 
with respect to the group of K-unitary transformations of rj. The definition of the 
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sine and cosine functions is given in Definition 3.1. Now the Grassmannian manifold 
G n , r is a compact Riemannian symmetric space G n>r = G/K, and the space C°°(G njr ) 
is decomposed under G into irreducible subspaces by the Cartan-Helgason theorem. 
The operators 7Z, C and S are G-invariant and hence they act on the irreducible 
subspaces as scalars, which can also be interpreted as the spectral symbols of the 
transforms. The spectral symbol of the Radon transform has been found earlier in 
[6] by integral computations using explicit formulas for highest weight vectors. In the 
present paper we shall find the spectral symbols of the cosine and sine transforms and a 
characterization of their images. We shall use the spherical transform on root systems 
instead of explicit formulas for highest weight vectors as in [6]. The same technique 
can also be used to derive some of the results of Grinberg [6]. Our work generalizes the 
recent results of [TJ on cosine transform where the case K = M and 2v — 1 is considered. 
As application we find further the unitary structure of the Stein's complementary series 
of the group GL(2r, K) realizing on the Grassmannian G2 r ,r- We give below a more 
precise description of our results. 

Firstly there is a close relation between the three transforms. The cosine and sine 
transforms can be factorized in terms of the Radon transform; see Lemma 3.4. The 
integral kernels |Cos(^, ?7o) | 2l/ and |Sin(£, rj )\ 2u , for r = r', at a base point i] are K- 
invariant functions and their restriction on the Cartan subspace A ■ rjo are the product 
of the usual cosine and respectively sine functions. Thus they are related roughly 
speaking by a rotation of |. On the other hand, the square R*, r R r > r for r' > r is 
an integral operator on G n ^ r and the integral kernel is actually the cosine kernel; see 
[22] for the case of non-compact symmetric matrix domains. Now there is a fourth 
transform, the Berezin transform which appears naturally in the branching rule of 
holomorphic representations and quantization; see [25], [26] and references therein. 
We prove also that the cosine transform on Grassmannians is the compact analogue of 
the Berezin transform [25] on non-compact symmetric spaces. As another application 
we find the branching rule of certain scalar holomorphic representations on the complex 
Grassmannian under isometric group of the real or quaternionic Grassmannian as a 
submanifold of the former; see Section 6.2. Thus the three transforms and the Berezin 
transform are related intrinsically and we can treat them with a unified approach. 

The sine transform S r>r is related to the unitary representations of a large non- 
compact group GL(2r,K). In |T9] Stein proved that there is a family of unitary ir- 
reducible representations of GL(2r, C) induced from non-unitary representations of a 
maximal parabolic subgroup, also called the Stein's complementary series. The uni- 
tarity of those representation was proved in [19] by computing the Fourier transform 
of the kernel |det(x)| a of the Knapp-Stein intertwining operator; it gives then the 
unitary structure in the so-called non-compact picture of the induced representations. 
Vogan [20] proved that the Stein's results can be extended to GL(2r, K) and proved 
the corresponding existence of the Stein's complementary series. We prove that the 
sine transform S r>r is the Knapp-Stein intertwining operator for the group GL(2r, K). 
We characterize the unitary structure of the Stein's complementary series in the com- 
pact picture; see Section 6.1. The unitarity and composition series for this family of 
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principal series representation of GL(n,K) are of considerable interests (see [TT], [T5] 
and [23]), and they have also applications in the theory of valuations on convex sets 

m. 

Here is an outline of our idea and method of computing the spectral symbol. As 
just mentiond the squared Radon transform 1Z*, r H r ',r, on a non-compact symmetric 
matrix domain has an integral kernel being a product of hyperbolic sine functions 
whereas [26] the Berezin transform has a product of hyperbolic cosine functions, both 
in terms of the geodesic coordinates. The spectral symbol of the Berezin transform is 
then the Harish-Chandra spherical transform of the hyperbolic cosine functions cosh 1 ' t. 
We find in [26] this spherical transform on a root system of type BC with general root 
multiplicities. This is done by deriving first certain Bernstein-Sato type formulas for 
hyperbolic cosine functions cosh 1 " t, performing spherical transform and then by using 
the observation that the functions cosh _iy t, v — > oo, tend to the deltat function at 
t = up to normalization constants. This idea has been used before and might be due 
to Berezin. Similar formulas for the hyperbolic sine functions is proved in [22J , and is a 
crucial step in finding the inversion formula for the Radon transform on non-compact 
symmetric domains. Using the techniques developed there we derive in this paper some 
corresponding Bernstein-Sato type formulas for the sine and cosine functions on the 
compact torus of a general root system of type BC using the Cherednik operators, and 
we obtain the Cherednik-Opdam transform for the sine and cosine transforms. This 
method can also be applied Radon transform, giving a different proof of some of the 
main results of Grinberg [6], but we will not present the details here. 

Our main results are stated in Theorems 14. 1[ I4.3[ 5.4 and 6.2. In Theorem 14.11 we 
present Bernstein-Sato type formulas for the cosine and sine functions, Theorem 14.31 
computes their spherical transforms and Theorem 5.4 gives a characterization of the 
images of the cosine and sine transforms. In Theorem 6.2 we give an independent 
proof for the existence the Stein's complementary series and provides a formula for 
their unitary structure. 

After this paper was finished I was informed by Professor Semyon Alesker that 
Theorem 5.4 (1) was partly proved by him in an unpublished preprint [2] (under the 
assumption that v is not a negative half-integer). The method there is quite different, 
it uses the localization theorem of Beilinson-Bernstein combined with some results 
by Braden- Grinberg on perverse sheaves and by Howe-Lee on the K-type structure 
of degenerate principal series representations. It is my pleasure to thank him for his 
comments on an earlier version of this paper, and for sending a copy of his preprint. 
I thank also Professor B. Rubin for informing me that a special case of Proposition 
5.1 (on the real Grassmannian when all rrij are equal) was also obtained in [17] as a 
consequence of Hecke equality and Gindikin Gamma functions. The expert comments 
on an earlier version of this paper by the referee are greatly acknowledged. 
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2. Grassmannian manifolds G n>r AND the irreducible decomposition of 

L 2 (G n 



' n,r I 



2.1. Grassmannian manifolds X = G n r as symmetric spaces. Let K — R, C,H 
be the field of real, complex or quaternionic numbers with the standard involution 
(conjugation) x — > x and let a = dim^E. = 1,2,4. Let M n ^ m := M„ ;m (K) be the space 
of all n x m-matrices, also viewed as K-linear transformations from K m to K n , where K 
acts on the right by scalar multiplication. Denote x* = x T , the conjugated transpose, 
for x G M n>m . Let 

G := U(n, K) = {ge M n , n ; g*g = I n } = 0(n), U(n), Sp(n) 

be the orthogonal, unitary, and symplectic groups accordingly. Denote, for x G M n>n , 
detR(a;) the determinant of a; as a real linear transformation on K n = R an . (For K = C 
det R (x) = | det c (x)| 2 and for K = H, det K (x) = dete(x) 4 where det H (^) is the so-called 
Dieudonne determinant.) 

Consider, for r < n, the Stiefel manifold S nj1 . of all orthonormal r-frames in K™, and 

X G nr 

the Grassmannian manifold of all r-dimensional subspaces over IK. S n ^ r can be realized 
as the set of all matrices x G M n>r such that x*x = I, namely K-linear isometric 
transformations x G K r — > K™. Each x G S njr defines uniquely a r-dimensional 
subspace ^ over K, 

^ = { x } ■= x K k C K n G G n , r . 

Thus X is identified with the space of orbits in S njr under the action of the unitary 
group U (r, K) on W, 

X = G n , r = S n , r /U(r,K). 

Let 



£o = K r © = {:r } G G„, r , x 







We will fix £ and xo as reference points of X and S^, The group G acts on S njr and 
by the defining action; they are then realized as a homogeneous and respectively 
symmetric space, 

(2.1) S ntr = G/U(n-r,K), X = G/K = U(n, K)/U(r, K) x U(n - r, K). 

Here K = U(r,K) x U(n — r, K) is the isotropic subgroup of consisting of block 
diagonal matrices of the form 



A 
D 



AeU(r,K), D G U(n — r,K). 



Due to the G-isometry between G n r and G ni „_ r we can and will assume in this 
paper, if nothing else is stated, that 

(2.2) 2r < n. 
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2.2. Irreducible decomposition of L 2 (X). To describe the irreducible decomposi- 
tion of L 2 (X) under G we let g be the Lie algebra of G and g = t + p the Cartan 
decomposition of g, where 6 is the Lie algebra of K. The linear subspace p consists of 
n x n matrices of the block form 



Px 



X 

X* 



X e M, n _ 



which will be identified with M rjn _ r via the mapping X \— > px- We let o be the linear 
subspace of p = M r n _ r consisting of matrices of the form 

X = [diag{ii, • • • ,t r } rjn _2r] = hE x H \-t r E r , t 1 ,---,t r e'R 

with Ej being the matrix having 1 on the (j,j) position and on the rest positions, 
j — 1, • • • , r. The maximal torus A = exp(a) is then the group of exp(X), with 



(2.3) 



exp(X) 



diag(cos£, 
diag(— sin t, 



, cost) diag(sint, 
, — sini) diag(cost, 











, sin t) 
, cost) 



written as block matrix under the decomposition of K n = W © W © K. n ~ 2r . Let 
g* = t + ip = t + \/— Tp, the non-compact dual of g. The root system of R(g*, id) is of 
the form 

R{g*,ia) = {±e j ± e k } U {±ej} U {±2^} 

with respective multiplicities a, a — 1 and a(n — 2r). Here {Ej} is the dual basis of 
{iEj}. It is of type C (if n = 2r, a > 1), type BC (if n > 2r, a > 1), type B (if n > 2r, 
a = 1), or type D (if n — 2r, a — 1). It is understood here and in Section 4 that roots 
with zero multiplicity will not appear, so that all types are considered as special case 
of type BC. We fix an ordering so that S\ > ■ • • > e r > (and the condition e r > is 
dropped for type D). 

The following result follows from the Cartan- Helgason theorem [TUJ. (Strictly speak- 
ing the theorem is for G/K with simply connected and semisimple Lie group G. The 
group G — U(n, K) for K = C or H is not semisimple, but the result can be reduced 
to the semisimple group SU(n, K). However for K = R the group G = 0(n) is not 
connected nor SO{n) is simple connected. This imposes more conditions on the highest 
weights than those given by that Theorem.) 

Proposition 2.1. Under the action of G = U(n,'K) the space L 2 (X) decomposes as 



(2.4) 



L 2 (X) =J>- 



with multiplicity free, where each V— has highest weight m 
being integers and 

nil > m 2 • • • > Tn T — 
for K = C, H or X = R with n - 2r > and 

mi > m 2 • • • > \m r \ 



vn,\E\ + ■ ■ • m r e r with 



(i 
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for K = R and n = 2r. Each V— has a unique i^-invariant function 0m, called spherical 
polynomial, normalized so that 0m (£o) = 1- 

We will view the spherical polynomial 0m on X as defined on a = R r via the 
exponential mapping a — > G n ,r, X — > exp(X) • £ , namely 

0m(*i, • • • , t r ) := 0m(exp(ti J B 1 H h t r £ r ) • 

Finally we recall the polar decomposition of G n>r . For any function / on G n , r we have 
the following formula (which in turn fixes a normalization of the invariant measure) 

(2.5) / /(O^ = / / f(ka) ■ ^ )dkdfj,(a) 

JG n , r J K J A 

where 

djji(a) = dfi R (t) = ] [ |2sina(z£))| ma Gfti ■ • -dt r , a = exp(t) = exp(t 1 £' 1 H \-t r E r ) 

with mo, the root multiplicity of a and dfc the normalized measure on X; see [TU1 
Chapter I, Theorem 5.10]. 

3. Radon, cosine and sine transforms 

3.1. Cosine and sine of angles between subspaces. We fix the standard Euclidean 
norm on W 1 = R an . For any convex subset S in a <i-dimensional real subspace of K" we 
let voLj(S') be the corresponding Euclidean volume. The following definition is given 
in [1] for the real Grassmannians. 

Definition 3.1. If r < r' the cosine of the angle between two subspaces £ G G n ^ r and 
r\ G G n y is defined by 

where is the orthogonal projection from K n onto 77 C K n and A C £ is any convex 
set of non-zero volume. If r < 77, — r' the sine of the angle between two planes £ G G n . r 
and 77 G Gn,r' is defined by 

|Sin(£,r/)| = |Cos(£V)|. 

For general r and r' we define |Cos(£, 77) | and | Sin(^", 77) | by the symmetry condition 

|Cos(e,77)| = |Cosfo,fl|, |Sin(e,77)| = |Sin(77,e)l- 

In [5] Grinberg and Rubin introduce also a cosine of angle, COS 2 (?/,x), between two 
elements x G S n>r and y G S n>r i, for r < r', defined to be the r x r-semi-positive definite 
matrix 

(3.2) COS 2 {y,x) = x*yy*x. 

The following lemma computes the cosine (3.1) in terms of (I3.2p 
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Lemma 3.2. (1) Let 1 < r < r' and the notations be as above. The two cosine 
functions (13.11) and (13. 2\i are related by, 

|Cos(7/,0| a = (det R COS 2 (y,x))i 

where rj = yW G G n y, £ = yW G G n ,r- 

(2) Let 1 < r = r' < n - r. Write £ G G„ ir as £ = A;exp(t 1 E 1 H h £ r £ r ) ■ £ , 

k E K. Then 

r 

(3.3) |Cos(e,eo)|' 5 = II I cos^l' 5 . 

r 

(3.4) |Sin(£,£o)| 5 = ni sint /- 

i=i 

Proof. We recall first that if T : M p — > M. q , p < q, is a linear transformation and -B C K p 
is any convex set of positive volume, then vol p (T(I?)) = det (T*T) 2 vol p (B), where T l 
is the transpose of T with respect to the Euclidean metrics in W and W 1 . Indeed, let 
T = [/(T*T)2 be the polar decomposition of T with U being a partial isometry. If 
rank(T) < p, then both vo\ p (T(B)) and det(T*T)a are zero and the formula is trivially 
true. If rank(T) = p then U t JJ — I and U is an isometry. We have then 

vol p (T(B)) = vol^C/^T)!^)) = vo] p ((T*T)3(B)) = det((T t T)5) vol p (B), 

proving the identity. 

Now if £ = {x} = xW and rj = {y} = yK r ' , with isometries x G S U;r and y G S n y, 
we have = yy*. Let A C £ = xW be any convex set of nonzero volume. We 
write A = x(B) with B C K r a convex set, and vol ra (I?) = vol m (v4) since x is an 
isometry. Its image under P v is P V {A) = yy*x(B). We apply the previous formula 
with T = P v x = yy*x, noticing that T*T = x*P^x = x*P n x = x*yy*x, 

vo\ ra P v (A) = (det M (x*yy*x))Kol ra (A) = det R (COS 2 (x, y))s vol ro (A). 

This proves the first part, and the second part is then a special case by using the 
formula Q. □ 

3.2. Factorization and diagonalization of the cosine and sine transforms. We 

define the Radon transform 1Z r i r : C°°(G njr ) — > C°°(G n y) by 

(3.5) (K r ,, r f){r))= [ f(0d v Z, 
if r < r' and 

(3-6) (^,r/)(l7) = K>,J(V) = [ 

if r > r'. Here is the unique Riemannian measure on the subset induced from 
the Riemannian measure on G n , r \ see [5] for an expression of TZ r > >r and the measure in 
terms of the realization (12.11) . In particular the Radon transform lZ r t >r commutes with 
the actions of G on C°°(G ntr ) and C°°(G n y). 
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Definition 3.3. Let 1 < r, r' < n and v > 0. We define the sine and cosine transforms 
from C°°(G ni r) to C(G n>r ,), by 

s { ;if(v)= [ \sm( V ,o\ 2u f(Od^ Om=/ |Cos(77,OrV(0^- 

The sine and cosine transforms are related to the Radon transform via the following 
Lemma. This Lemma in the case when 2v = 1 and IK = M is known and is proved 
in Lemma 1.7 in pp. The same method can be applied to the present case for general 
v > 0, by using a variant of the Cauchy-Kubota formula 



vo\ ar (B) s = c s [ vol ar (Pr e B) s dt 
J tec, „ 



for any convex polygon in 5 C (o, which follows easily by the invariance of both sides 
under translations and linear actions on £o- We skip the elementary proof. 

Lemma 3.4. Let 1 < r < r' < n, and v > 0. The cosine and sine transforms can be 
factorized as 

where C\ = c\{v) and C2 = C2{v) are some positive constants. 

The constant c% and C2 can be computed by using the integral formula in [2T] but 
we will not need it here. 

By taking conjugate of the above formulas we get factorizations for any 1 < r, r' < n, 
noticing that (C#)* = C { fi r , (5$)* = S™. 

The transforms dfli and S^, clearly intertwine the action of G. Consider the cor- 
responding decomposition of L 2 (G n y) according to Proposition 2.1, 

e 

(3.7) L 2 {G ny ) = ^W^. 

m 

Thus dfl, and Sfl> are diagonal operators up to a normalization. The eigenvalue of 
TZ r y has been found earlier by Grinberg [6]; see Theorem 5.2 below. In view of Lemma 
3.4 above, to find the eigenvalues of Cffi and S^y we need only consider the case when 

Denote c„(m) = c„ >r (m) = and s u (m) the eigenvalue of df) and respectively S'f) 
on V— . They can be evaluated by 

(3.8) CtVm(6) = c,(m)0 m (e o ) = C(m). 
Namely 

(3.9) c i ,(m) = |Co^)| 2 -(m)= f |Cos(£, e )| 2 ^m(0^, 

is the spherical transform of |Cos(-, £o)| 2l/ - Similarly for s„(m). They are integrals 
of ^-invariant functions. We use the polar coordinates f)2.5p . which further can be 
written as an integral on the quotient W /2nQ v of W by the (spherical) coroots lattice 
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Q v , namely the lattice generated by ct v = t^-t for a G R, where a is the dual of a, 
A(d) = (A, a); see [HH Chapter I, Theorem 5.10]. 

Lemma 3.5. Denote 



3 I ' 



|Cos(t)| := | J^Jcostjl, | Sin (t) | := | J^sintj 

3=1 3=1 

with 

t = hEi + ■■■ + t r E r = (t u ■ ■ ■ , U) e R r /2nQ v . 

Let v > 0. The eigenvalues c„(m) and s^(m) of the cosine respectively sine transforms 
are given by the spherical transforms 

(3.10) c(m) = |Co^ 2 -(m) = f | Cos(t) l 2 ^^, • • • ,t r )dfi R (t), 

(3.11) s„(m) = ISh^l^Gm) = / |Sin(t)| 2 > m (t 1? • • • ,t r )dfi R (t). 

4. Spherical transform on compact torus associated with general 

root system of type bc 

4.1. Bernstein-Sato type formulas for cosine and sine functions. We consider 
the root system R on a Euclidean space ia = iW (i = y/—l) of type BC 

(4.1) R = {±ej ±e k , j ^k}U {±s k } U {±2e k } 

with general non-negative multiplicities a, 2b and l for the respective sets of roots. Here 
{6j} is the dual basis in (ia)* of a fixed orthonormal basis {iEj} of ia; the notation 
{Ej} here coincides with that in Section 2.2. We use the same convention there, so 
that type D and type B will be considered as special cases. 

Denote by W the Weyl group. We will compute the spherical transform of certain 
Weyl group invariant sine and cosine functions, by using the harmonic analysis of the 
Cherednik operators developed by Opdam [16J. We follow the presentation there loc. 
tit., however with our roots being twice of the roots there and our multiplicities half 
of the ones there. 

We let Dj = D iEj , j = 1 • • • , r, be the trigonometric Cherednik operators acting on 
functions on W/2irQ v , 



= dj — ia -77- — — su-i) + ia> ^-n — rr(l — s ik) + 

3 Z_-f l _ e -2i(t h -tj) v K 3> l _ e -2t{tj-t k ) V 3KJ 

k<j j<k 



1 

1 — g—'Mtj 



+ 2i h —2At-S X -°3)- % P3: 



10 GENKAI ZHANG 



where Skj, cr k j and Oj are the reflections corresponding to the roots Ej — e^, ej + £fc,and 
Ej. Here 

1 r a 

P = g £ m « a = £ P? £ i> Pi = ^ + & + 2 ( r - J')- 

is the half sum of positive roots, m a being the root multiplicities. Let 0m be the 
Heckman-Opdam Jacobi polynomials on the root system ([8], [7], and [IE]), and 

Km) ■= [ f(t)<f> m (t)dfi R (t) 

be the spherical (or Jacobi) transform. Here Q y is as before the (spherical) coroot 
lattice. Our objective is to find the spherical transform of the functions | Cos(t) | 2l/ 
and |Sin(t)| 2l/ in Lemma 3.5. We establish first certain Bernstein-Sato type formulas, 
more exactly we will find certain Weyl group invariant polynomials of the Cherednik 
operators mapping | Cost | 5 to | Cost | 5-2 . 

Theorem 4.1. Let 5 > and Ais be the operator 

r 

M s :=U(D 2 + (6 + Pl ) 2 ). 

3=1 

Then the following Bernstein-Sato type formulas hold, 

r 

M s \Cos(t)\ 6 = Y[(S + a(j - 1))(5 + L-l + a(r- j))|Cos(t)| 5 - 2 
i=i 

and 

r 

Ma|Sin(t)| 5 = Y[(5 + a(j - 1))(5 + i + 2b-l + a(r- j))|Sin(t)| 5 - 2 . 

3=1 

Proof. In [26l Theorem 2.1] and [22j Theorem 3.1] the following formulas are proved 
for the hyperbolic sine and cosine functions, defined on ia, 

r r 

i=i j=i 

r r 

= Y[(5 + a{j -5- L-a{r-j))(Y[coshxj) 5 - 2 , x = {x ir -- ,x r ) G W, 

i=i j=i 



r 



n((^) 2 -(5+pi) 2 )(ii sinh 

3=1 3=1 



Xj , 



= Y[(5 + a{] -l))(5-l + L + 2b + a{r-]))([[smhx j ) 5 - 2 , x = (x u --- ,x r ) el r + , 

3=1 3=1 

where iDj is acting on the variable x. The function (sinha;) 5 and (cosh a;)" 5 are analytic 
functions on the right hand plane > 0, so are the products J7 t ^ =1 (sinhx 7 )' 5 and 
n^ = i(coshx :( ) 5 on the product of the right half plane {x; $t,Xj > 0}. All the identities 
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has a limit at the points x = it, t e W , £j 7^ 0, |. Our result follows then by taking 
the limit, observing also that sinh 2 (itj) = (—1) sin 2 £. □ 

Remark 4.2. The above theorem can also be proved directly by a straightforward but 
long computations. First, we can choose a dense open subset of the orbit W\W /2itQ v 
of the Weyl group so that the functions sint, > 0, cost,- > 0. Furthermore the operator 
M.s can be factorized as follows, 

r r 

(4.2) M 6 = + (6 + Pl f) = + 0* + Pi))(iDi + ($ + Pi)) 

i=i i=i 
and we may compute successively the action of each factors. We have, for each fixed 
h 

3 3 / 3 -n 

Y[(iDi + (5 + Pl ))Cos(£) 5 = l[(5 + a(l - 1)) Cos(£) 5 J] — 

\ COS T/i 

\ 1=1 . 



1=1 



i=j \ i=i 1 ' 



1 e~ itl \ -r-T e itl 



=n(«-i+.+«(r-o) cost^n^ in OK ,, 

i=j \ (=i / J=j 

fl(*A + (* + Pi))Sin(t) 5 = V f[(5 + a(l - 1)) ( Sin(£) 5 f[ ^- 

i=i i=i V 1=1 111 ' 



J 



-it; 



nHA+(5+px)) sin(t)*n-, 

1=3 \ 1=1 



^n^-i+^+^+^-o) sm(^n-- n- 



sin ti J sin ti 

i=j \ i=i V i=j 



by similar computations as in [2S] and [22] , which together with the factorization (14. 2 p 
implies our theorem. (The pattern of the identies is roughtly that each action of 

it ■ 

(iDj + (5 + pi)) on the funtions Cost produces an extra factor -^-j-, further action by 
(iDj + (5 + pi)) produces an extra factor ^j-', the e ±lfj cancels and we get a singel 

factor of Cos _2 £, namely our formula.) The above family of identities is to be compared 
with the trivial trigonometric formulas 

d e~ u 

(i— + 5) cos(t) 5 = 5cos(£) 5 , 

at cos t 

A / p-it \ / p-it \ p it 

{-i- + 5) cos(£) 5 — - ) = {5 - 1) cos(£) 5 N 



dt \ cos t J \ cos £ / cos t 

<l + 6) sin(t) 5 = iS (sm(t) 5 - * 



dt \ sint 

d / p _i< \ / p~ it \ p u 

-i- + 5) sin(t) 5 — - )=(-<)(*-!) sin(£)*- 



(it \ sin £ / \ sin t J sin £ 
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It would be interesting to reformulate the identities systematically in terms of Hecke 
algebras [3]. 

4.2. Spherical transform for cosine and sine functions. We let N u and N' v be 

the following normalization constants, 

(4.3) N v = I |Cos| 2 " \t)dn{t) 

jR r /2nQ v 

and 

(4.4) K= [ \Sm\ 2u (t)dfi(t). 

JR r /2ttQ v 

Their exact values can be evaluated by using the Macdonald formula for generalized 
Beta-integrals (see [151 Ex. 7, Sect. 10, Chapt. VII]), 

at _ oar(r-l)+2r6+2rv! IT Eit^ ~ ! + 

11 r(f(j-») 

l<i<j<r v 2^ ' 

T a (l + 5 + ^ + f (r - 1))T> + 1 + + f (r - 1)) 
T a (v + 1 + 6 + l + a(r - 1)) 

at-/ _ 9 or(r-l)+2r6+2rt_! TT Eifjj ZllLll v 

" 11 r(f(j-0 

l<i<J<r V2VJ ' 

r«(i + ^ + f (r - l))r a (z/ + 1 + + f (r - 1)) 



(4.5) 



(4.6) 



r(i/ + l + 6 + t + a(r-l)-|0'-l)) 
Here T a (a) is the Gindikin's Gamma function 

r 

r a (a) = n r («-^0'- 1 ))- 

We recall also the Pochammer symbol (u)k = {y)(y + 1) • • • [y + k — 1). 

Theorem 4.3. The spherical transforms of the functions |Cos(t) | 2zy and | Sin(t) | 2t/ are 
given by 

JL (^+1 + 10-1)-?) = 

C, r (m) : = ICosl-(m) = tf. TJ (|) + j + ; + „ + a(r _ T) _ . l}) 



and 



3=1 " — 



^ + l + f(j-l)-^H 



L ( I/ + i + t + 6 + a ( r _i)_| ( -_ 1)) ^ 2 ,-- - , T 

1 Z 2 



s„, r (m) := |Sin| 2 "(m) = N' u J] , ; ., , , ■ ^ -. . 



For root systems of Type C or Type D we have 0m(f , ■ • • , f ) = 1 and thus 

(-„-§(,•-!)); 



:= |Sm| " (!!i) = <n (, +1+t+6+a ( r -i)-g (J -i)) 
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We need the following elementary result, which states simply that the normalized 
integration of the cosine functions cos n t and sin n t tends to the 5-function at t — 
respectively t — f . 

Lemma 4.4. Suppose be a bounded and continuous function on W/2ttQ v . Then 

lim -L [ |Cos| 2 ^)0( S )^(s) = 0(0) 



and 

2'"' '2 



lim -L / iSinHs)^) 



The following lemma asserts that 0m(§ , ■ • • , f ) is always nonzero, which is needed 
in Theorem 5.4. 

Lemma 4.5. (1) Suppose i? is a root system of type C, or D with general non- 
negative root multiplicity. Then 



,7T 71", 

2 : 



'ill 1 ' | [ 



3=1 

(2) Suppose i? is the root system of the Grassmannian manifold G njr . Then 

0m(-,---,2)^ - 

Proof. (1) From the formula for Dj we see that, if R is a root system of type C, or D, 
then {Dj} are invariant under the map (ti, ■ ■ ■ ,t r ) i— ► (ti + f , • • • , i r + |). Moreover 
it is easy to prove that the polynomial f(ti, -,t r ) := 4>aJti + f > " " >^r + f) * s a ^ so 
invariant under the Weyl group. We prove this for type C, the type D is the same. 
The Weyl group is generated by the simple reflections a r and Sjj+i, so we need only 
to check the invariance for those elements. Notice that by definition m is invariant 
under the mapping (t\, ■ ■ ■ ,t r ) i— > (ti, - • • , t r + 7r). We have 

7T 7T 

(<Trf)(tl, ■■■ ,t r ) = f(h, ■■■ , -t r ) = + 2' " ' ' ~ f r + g) 

= 0m(*l + 7T> ' ' ' ' ~( t r 



7T 



2' ' w 2' 

= 0m(tl + |,-" ,~(tr + |)) 

= 0m(tl + ,(t r + -)) 

= /(*1> " " " ,t r ) 

and that Sjj + if = f is trivially true. Thus / is an eigenfunction of the Weyl group 
invariant polynomials of the operators {Dj} with the eigenvalues being the same as 
that of 0m- Namely f(t) = c0 m (t) for some constant c, by the uniqueness of the 
spherical eigenfunctions [16J. Comparing the leading coefficients we find the constant 
c, and the evaluation of / at t = proves our result. 
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(2) Consider the case of Grassmannian manifolds. We need only to treat the case 
when the corresponding root system R is of type B or BC, namely G n . r (lK) for n—r > r. 
Write 

£1 := exp(|#) • £o = © K r © G G n ,, n 

where exp(^E) is the short-hand notation exp(|_E) = exp^Ei H — • + f We have, 
by the integral formula for spherical polynomials, [10J Chapter IV, Proposition 2.2], 
that 

, 2 f , , ,7T ... , - 



(4.7) <f>m(ti) = J K ^rn(exp(-E)kexp(-E) ■ ^ )dk, 

which is further an integration on a subset of G n>T . We claim the set of this integration, 
namely 

(4.8) S := {£ = exp(^)A;exp(^) ■ £ ; k G K}, 
is given by 

5 = G G n ,r;£C 77x } 

where r/i G Gn, n -r is the element 

r)i := K r © © K n - 2r C K r © K r © K n " 2r = K". 

Indeed let £ = exp(^E)k exp(|£') • £ be any element in the set S for some k G K. 
Let fc = diag(v4, D) £ U(r) x U(n — r) and write D as a 2 x 2-block matrix under the 
decomposition of K n ~ r = K r © K n ~ 2r , 



D 2 i D 2 2 



Using the formula (I2.3P for the exponential we find that £ is of the form 

£ = {D u v © © D 21 v; v G K r } C r] X . 

Conversely, it is rather elementary to see that any r-dimensional subspace of 771 is of 
the above form for some D. This proves our claim. Therefore, the integration (14. 7p is 
precisely the Radon transform of (f> m , namely 

0m (£l) 2 = Tln-r^miVl)- 

Now the function TZ n - r ^4>m is a K- invariant function on G„ in _. r , and its dual Radon 
transform evaluated at £0, ^n-rr(^n-r,^a)(6) = ^r,n-r(^n-r,r^m)((o) is an inte- 
gration of TZ n -r, r (pm over the set {r];r] D £ } which is a K-orbit of 771. Namely the 
integrand is constant and 

Putting those together we have 

0m(6) 2 = R-n-r, r <l>nJj}l) = ^^A-r^mKo) • 

It follows from the main result in [6] (see Theorem 5.2 below) that 7?4_ rr 7£ n _ r!r 0m(£o) 
is a non-zero constant of 0m(£o) = 1 since r < n — r. Thus 0m(£i) 7^ 0. □ 
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Remark 4.6. One may follow the argument in the proof and find the constant m (£i) 2 
by using the result of Grinberg. It would be interesting to evaluate the constant for 
a general root system of positive multiplicities. For the real projective space G n> i = 
P n_1 (]R) the second claim (2) is proved in [TQl Chapter I, Lemma 4.9]. 

We prove now Theorem 4.3. 

Proof. We perform the spherical transform on the Bernstein-Sato formula for the cosine 
function in Theorem 14 . 1 1 with 5 = 2z/+2. Using the self-adjoint property of the operator 
M.2u and that (see |16|), 



-M 2i ,+20m = Y[((2v + 2 + pi) 2 - {m 3 + Pjf)(j) T _ 

3=1 



we find that (suppressing the subindex r in c 1J>r (m) ) 

-c u+1 (m). 



c„(m) _ N u+1 n;=i((2^ + 2 + Pl f - K + Pj f) 1 



Nu " N v Yl] =l (2v + 2 + a( 3 -l))(2v + l + L + a(r-j))N v+l 
After a simplification we get 

/ x r nij rrij , \ 



II( 1 - ^-TxfaT )( 1 + 



JV„ ~ ^ z/ + l + f(j-l) ;v z/ + l + 6 + i + a(r-l)-f(j-l) ; 
Iterating the result produces furthermore 

r I rrii 



c u {m) c^i+i^raj 



iV, A^+i Jlir z/ + fc+l + f(j-l] 



z/ + fc + 1 + 6 + 4 + a(r - 1) - § (j - l) 7 ' 
However Cu + l+1 ^ m (O) = 1, Z — > oo, according to Lemma 4.4. Therefore, 



v + l + \ 



N » * + * + i + §(;-i) 



(1 + i/+fc + l + 6 + t+V-l)-fO'-l)^' 
which can also be written in terms of the Gamma function ([H p. 5]) 

T(v + 1 + |(j - l))r(i/ + 1 + b + 1 + a(r - 1) - |(j - 1)) 



n 



Arfi/ + l + af.,'_ 



11 T(z/ + 1 + f (j - 1) - ^-)Y{u + 1 + 6 + i + a(r - 1) - f (j - 1) + 
This proves our formula for the cosine function. The sine function is done similarly. □ 
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5. Spectral symbols and range characterization 

5.1. Diagonalization of the transforms. Theorem 4.3 applied to Lemma 3.5 gives 
then 

Proposition 5.1. Let K = R, C or H, and 1 < r < n — 1. The eigenvalue of Cr,r and 
<Sr,r are given by c 1/r (m) and s, //r (m) in Theorem 4.3 with i = a — 1, b = a(n — 2r). 

To state our result on the spectral symbol of df], and S'f}, for different r and r' we 
recall first the following result of Grinberg [B], reformulated slightly differently here. 

Theorem 5.2. Let E = 1, C or H, and l<r<r'<n — 1. Then the operator 7£ r / i? . 
defines a bounded operator from L 2 (G n . r ) into L 2 (G n , r i). The operator 1Z*, r lZ r ^ r is a 
diagonal operator under the decomposition, 

H%Hr»,rf = c(m)/, / e v— , 

with an explicit formula for the eigenvalue c(m). The closure in L 2 (G niT >) of the image 
of the operator 1Z r i r on L 2 (G n)r ) is 

raeL rr / 

where L r y is the subset of those m for which rrij — if j > min{r, r'}. 

Note that the L 2 -bounded result was not stated in [BJ. However it follows directly 
from the explicit formula for the eigenvalue c(m) there. We remark also that the 
explicit formula for c(m) found in |6j is under the condition r < r' and 2r' < n. For 
general r and r' the formula for c(m) is given in [121 Theorem 6.4]. 

Using Lemma 3.4, Proposition 5.1 and Theorem 5.2 we get 

Corollary 5.3. Suppose v > 0. Let 1 < r, r' < n— 1. The eigenvalue of C* r ,C r y and 
S* r ,S r y on the space V^— are given respectively by 

c(m) c^,. (m) c^ r ' (m) , s (m) s VjT (m) s^,.' (m) , 

where c u<r is given in Proposition 5.1 and c(m) in Theorem 5.2 and [12] . In particular 
Q, r T > and S T T > are bounded operators from L 2 {G n y) to L 2 {G n ^). 

5.2. Characterization of the image of the transforms. The following theorem 
follows immediately from Corollary 5.3 and Lemma 4.5. 

Theorem 5.4. Let 1 < r, r' < n — 1 and v > 0. 

(1) Let K — R. If v |. Then the closures of images of C r ^ r and S r ', r , L 2 (G n ^ r ) — > 
L 2 (G n y), are given by 

(5.i) Yl w -- 

where L r y is given in Theorem 5.2. In particular the images are dense if r' < r. 
If v E | then the closures of their images in L 2 (G n y) are given by 
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where L v is the subset of m such that 



Tfl ' I 1 

(5.2) — °- < + 1), if + 1) is an integer, j = 1, 



r. 



(2) Let K = C or H. Mv^Z. Then the closure of the images are as in ( 15. ip . If 
i/GZ then the closures are given by 



where is the subset of m such that 



(5.3) 



m,- a . 

-^<z,+ l + -(j-l), j = l, 



6. The sine-transform as Knapp-Stein intertwining operator. Cosine 
transform and branching of holomorphic representations 

In this section we give applications of our results to the existence of the Stein's 
complementary series and on the branching rule of holomorphic representations. 

6.1. Stein's complementary series. We fix n = 2r in this subsection. Let GL n = 
GL(n, IK) be the general linear group over IK and G = U(n, IK) as before. Consider the 
parabolic subgroup P of GL n consisting of block matrices of the form 



V 



B C 
D 



where B,C,D e M r , r (K). The Langlands decomposition of P is P = LN = MAN, 
with the nilpotent group iV and its opposite N consisting of upper respectively lower 
triangular matrices 



(6.1) 



Tlx 



I X 
/ 



G N, n Y 



I 
Y I 



G N, X,Y eM r 



both being identified with M rr , the group L = MA consists of diagonal matrices 



V 



B 
D 



with B,D G GL r and A being a one-dimensional subgroup in the center of L. 
Let St, be the one-dimensional representation 



S t (p) = | det R (BD 



-lwt 



of P, for t E C. The special case Si. corresponds the determinant of adjoint represen- 
tation of P on N (written generally as e p where p is half-sum of positive roots with 
respect to the one-dimensional Lie algebra of A). 

Let Indp L "(t) be the induced representation [131 Chapter VII] consisting of measur- 
able functions / on G such that 



f(gp) = (5t + rJp))- 1 f(g) 
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and that f\ G G L 2 (G). When realized on N = M r ^ r = M rjf .(K) the Knapp-Stein 
intertwining operator from Ind P Ln (t) to Jn<ip Ln (— t) is given by [T9] . 



^(nx) = / I det R (X - Y)\(- r+ ^F(n Y )dY. 

We will use however the so-called compact realization of the induced representation, 
namely as acting the L 2 (GL n / GL n D L) of the compact manifold GL n /GL n n L. Now 
X = GL n /GL n fl L = G/K is precisely the Grassmannian manifold A" = G njr . The 
relation between the two realizations is given by a change of variables, 

(6.2) /(0 G L 2 (*) - F(n) = 5 t+s (p(n))-V(«(n)eo) 

where n = k(n)p(n) is the Iwasawa GL n = GP decomposition of n in GL n \ see [131 
Chapter VII] (our GL n and G correspond to G respectively K there). 

Lemma 6.1. In the compact picture the Knapp-Stein (formal) intertwining operator 
X t is given by (up to a non-zero constant) 



Gn 



J t f(0= I \Sin(tv)\ 2u f(v)dv, 

where 
(6.3) 



a , 



It is well-defined on the space L 2 (X) for v > 0. 

Proof. Let 77) denote temporarily the kernel of the intertwining operator in the 
compact realization on L 2 (X), namely let 



J t f(0= [ s(^ v )f( v )d v . 



The kernel S(£, rj) is then uniquely determined by S(£o, 77) (recalling that £ = JK r ©0 G 
X) by the transitivity of G = U(n, K) on X and by the intertwining property. The 
evaluation Jtf(£,o) can also be computed via the intertwining operator T t in the non- 
compact picture above, 

(6.4) I S(Zo,v)f(v)dv = Jtfiio) = TtF(0) = f | det R (r) | ^+ 2 *) F(n Y )dY 

where the function / on X and F on N is given by (I6.2p . Performing the change of 
variable Y G M r ^ r 1— ► 77 := k(ny)Co G in the integral over M r>r , according to (16.21) . 
we find that 

S(£ ,rj) = \det R (Y)t r+2t ^ t+ ,(p(n Y ))-\j a c Y ^y 1 , 
where Jacy^r? is the Jacobian of Y — » rj, namely 

drj = J&CYy~*r)dY. 
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We will find the two quantities £ i+ r(p(ny)) and the Jacobian. Consider therefore 
the GL n = GMAN decomposition (note that G is the maximal compact subgroup of 
GL n ) of n Y G N = M r , r (K) (under the identification ([63]) ) 



(6.5) riy 



J r 



k(n Y )p(n Y ), k(n Y ) G G, p(ny) 



B C 
D 



G P = MAN. 



This is a factorization of the lower triangular matrix n Y as a product of a unitary 
matrix k(n Y ) and an upper triangular matrix p(n Y ). Let the above element act on the 
base element (o£^ via the defining action of GL n acting on subspaces of K n . Since 
p(n Y ) stabilize £ we see that the change of variables Y — > 77 = k(n Y )^o G X is given 
by 

'' ° £0 = {v®Yv-v G K r } G 



(6.6) 77 = k(n Y )£ 



Y I 



An easy matrix computation then shows also that 

^ +i (p(ny)) = det M (l + r*F)* + i 

Indeed taking the determinant of (16.51) we find that det R B det R D = 1; computing the 
upper left entry of the block matrix n^ny we get B*B = I + F*y and 5 s (p(ny)) = 
det^BD- 1 ) 3 = det R (B) 2s = det R (B*B) s = det R (J + Y*Y) S for any s. 
The measure ^77, and equivalently the Jacobian, is given by 

dr] = det R (J + Y*Y)- r dY, 

by direct computations (or by using Proposition 3.3 (ii) in plj). Now the sine Sin(?7, £ ) 
of the angle between 77 and ^0 is, by (16. 6j) and Lemma 3.2, given by 

i G . ic \\a I det R (y)| 

det R (l + r*F)|2 

It follows finally that 

S(Z , V ) = \Sm(Z ,r ] )\-^- 2t \ 
as claimed. This completes the proof. □ 

Remark 6.2. Consider the simplest case when GL n = GL,2(M.). The compact picture 
of the induced representation in question is on the half unit circle S 1 /Z 2 = {e ie ,0 < 
9 < 7r} represent at ing lines M(cos#, sin^) in R 2 . The intertwining operator in the 
non-compact picture is an integral operator defined by the convolution by \x\ s on 
R. The change of variables from non-compact to the compact is then x — > M(l,x), 
namely x — > M(cos 9, sin 9), with I sin 01 = — The above lemma states that the 

(l+z 2 )2 

intertwining operator in the compact picture is then the convolution on the circle by 
I sin^l* = (||1 — e 2e \) s , namely it has the kernel |e 2 * — e 2e \ s , which is well-known. 

By using Lemma 3.5 and Theorem 4.3 we obtain then the eigenvalues of the Knapp- 
Stein intertwining operator J t on L 2 {G n ^ r ) under the decomposition (2.4), and we 
can determine the range of t for which the eigenvalues are all non-negative (or all non- 
positive). This gives an independent proof of the existence proof Stein's complementary 
series and with explicit formula for the corresponding unitary structure. 
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Theorem 6.3. Let v = — |(r — 2t). The Knapp-Stein intertwining operator J~ t , is 
well-defined for v > on the algebraic span of the subspace V— . It intertwines the 
actions of g[(n, K) by the induced representations Indp in (t) and Indp Ln (—t), and has 
meromorphic continuation in v. Its eigenvalues are given by 

r (| (r + l_ j _2t))™ i 

(6-7) ^11 717 



The sesqui-linear form 



for 



{f,g)u ■= -w{Jtf,g) 

V 

0<i<i 



is well-defined, and is a £j[(n, K)-invariant positive definite Hermitian inner product. 
The completion of the pre-Hilbert space is a unitary representation of of GL(n, K). 

For more details on unitarity and composition series of the whole family of the 
induced representations see also [11], [18] and 



6.2. Branching of holomorphic representations. Finally we give an application 
of our result to the branching of holomorphic representations on compact Hermitian 
symmetric spaces. The non-compact case has been studied intensively; see [25] and 
references therein. The result below can be deduced from the general theory [T3] 
of (discrete) branching of highest weight representations. We only indicate here a 
concrete approach using the cosine transform. To keep the presentation of paper 
rather explicit we will only treat the case when the Hermitian symmetric spaces is 
the complex Grassmannian manifold with real form being the real or quaternionic 
Grassmannians. 

We fix the complex Grassmannian manifold X\ of ri-dimensional complex subspace 
in C 2ri+bl , n > l,6i > 0, namely 

X x = U(2n + bx)/U{r x ) x U(n + 6i). 

X\ is equipped with the JJ{2r\ + 6i)-invariant Hermitian metric. Consider the real 
Grassmannian X, 

X = G/K = 0(2r + b) /0(r) xO(r + b), r = n 
or the quaternionic Grassmannian 

X = G/K = Sp(r + b)/Sp(r) x Sp(b), r = J' b =J 

when n and b\ are even integers. Then X can be realized as a totally geodesic real 
form of the Hermitian symmetric space X\. We realize the space V\ = M ri+bl , ri (C) as 
a dense subset of X\ by the identification 

z E Vi ^ {y © zy; y E C ri } G X x \ 

similarly V = M r -(-b >r (K) can be realized as a dense subset of X via 

z E V h-> {y © zy; y G K r } G X. 
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For any positive integer a there is a corresponding weighted Bergman space on the 
compact space X\, denoted by H a (Xi), with the reproducing kernel det(7 + w*z) a , 
z,w G V C X\, which forms an irreducible unitary representation of U(2r\ + b\). The 
elements in the space H a (Xi) will be identified with holomorphic polynomials on V\. 
Consider H a (Xi) as a unitary representation of G C U{2r\ + &i) and we will find the 
explicit irreducible decomposition. 

We can first realize the L 2 (X) as a space of functions on the subset V. Indeed the 
L 2 (X) is G-equivalent to 

L 2 (V, det(J + x*x)~^ tL dm(x)) 

where dm(x) is the Lebesgue measure. 
The restriction mapping 

T : H a (Xi) -> C^iX), f ^ f(x) det(J + x*x)~* 

defines then an intertwining map, where G acts on C° C '{X) by the defining action. The 
operator TT* on L 2 (X) = L 2 (V, det(I + x*x) 2 dm(x)) will be called the Berezin 
transform and is of the form 

la. ex. 2n + bi 

TT*f(x) = / det(J+x*x)-2 det(/+a;* 2 /) a /(|/)det(/+ 2 /* 2 /)-2 det(I+y*y) —dm(y); 

Jv 

see [25]. This is just the cosine transform, and its eigenvalues on the space L 2 (X) 
implies the branching rule of the representation of U (2ri + b{) under G. The first part 
of the next Proposition follows by similar computation as in the previous subsection, 
the second part follows from Theorem 5.4 and some abstract arguments. 

Proposition 6.4. (1) The Berezin transform TT* on X = G/K = G n , r (K) is the 
cosine transform Cr|r with v — % for K = R and v = a for K = H. 
(2) The representation TC a (Xi) is decomposed under G with multiplicity free as 

m£L„ 

where L u is given in (15.21) and (15. 3ft . 
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